A number of techniques exist in the art for testing systems, such as computer systems running machine executable code. By way of overview, a general class of these techniques operates by applying a predefined test to a System Under Test (SUT), generating a singular output result, and then comparing this singular output result with a singular expected result. The tester may designate an output result as anomalous if it diverges from an expected result by more than a predefined amount. In data driven testing, a tester may repeat a test operation for a range of input values to produce a corresponding range of output results. These output results are then pair-wise compared with associated expected results.
In one case, a tester may specifically configure a test to examine the functionality of the SUT. A functional test determines whether the SUT is generating an expected result (that is, whether it performs correctly or not). In another case, a tester may configure a test to examine the performance of the SUT. A performance-based test determines whether the SUT is running in a desired manner (that is, whether it is running quickly enough, utilizing proper amounts of memory or file space during execution, and so forth).
FIG. 1 depicts the above-described known testing strategy. In this figure, testing logic 102 applies a predefined test to a SUT 104. This operation generates an actual output result 106. Comparison logic 108 compares the actual output result 106 with an expected result 110 to generate a test outcome 112.
The above-described testing strategy often fails to adequately meet the challenges presented in today's technical environments. For instance, a tester will often need to sequence through a great number of tests to ensure that different functional and performance-related aspects of the SUT 104 are working properly. This requires tedious and time-consuming retooling of the testing logic 102 to sequentially apply the series of tests to the SUT 104. For example, some contemporary software-driven systems can provide different user interface (UI) presentations in a number of different natural languages. A thorough testing regimen therefore needs to test the same SUT 104 in different configurations associated with different languages. Again, this requires the burdensome task of devising different tests to run on the testing logic 102, applying these tests to the SUT 104 in different respective configurations, producing a series of corresponding actual results 106, and efficiently managing the myriad of results 106.
Further, the above-described testing strategy does not readily allow a tester to revise a testing strategy after a test has been run. For example, a tester may be required to perform a battery of tests in a fixed time schedule to meet an anticipated product release date. Each test may require a defined setup time and execution time. After performing a series of tests, the tester may discover that certain assumptions made during prior tests were not optimally suited to produce results that are needed. In this case, the tester must completely repeat the test, requiring a potential delay in the overall testing regimen.
Other types of testing strategies provide output results that convey more information than simply an indication of “Pass” and “Fail.” For instance, so-called multiple-outcome testing classifies a failed test as either “test known” or “test unknown.” Such a test will generate a “test known” state when it is able to ascertain the cause of failure; it will generate a “test unknown” state otherwise. However, this strategy does not solve the above-described problems, as it still requires a tester to sequence through multiple separate tests to fully explore the behavior of a SUT, and potentially completely repeat a test if prior testing assumptions were later determined to be inadequate.
There are additional drawbacks to the conventional strategies described above.
As such, there is an exemplary need in the art for a more efficient technique for testing systems, such as, but not limited to, software-driven computing systems.